Grammar frequency and simplification: when intuition fails

We investigate whether a medical writer can simplify text by only changing the grammatical structure. Based on a user study, we find that while the sentences look simpler after simplification, they are not easier to understand. For grammatical simplification, better tools are needed to provide more concrete guidance and feedback. Introduction Providing text to patients and health information consumers that facilitates comprehension helps create a healthliterate patient group. Over the last decades, readability formulas have been touted as writing support tools, but evidence shows they are inefficient and ineffective [1]. We are systematically examining different text features for their potential for simplifying text. We measure the prevalence of each feature, their relationship to text difficulty and how they can be used to simplify text. In previous work, we demonstrated strong results with term frequency, noun phrase complexity, and grammar frequency. In this paper, we examine grammatical simplification. Methods To evaluate sentence difficulty based on grammatical structure, we parsed all sentences in English Wikipedia and counted the frequency of the 3 level in the parse tree (which we denote the grammar frequency). In earlier work, we found that grammar frequency is indicative of sentence difficulty, even when controlling for other variables. We randomly selected 220 sentences from 11 grammar frequency bins (10 per bin) representing increasingly difficult grammatical structures. The writer was told to simplify each sentence by changing only the grammatical structure, i.e. not changing words to simpler variants. We evaluated the simplicity of the sentences before and after simplification with a user study measuring two metrics: perceived difficulty, measured on a 5-point Likert scale, and actual difficulty, measured using a multiple choice Cloze test over four blanked nouns in the sentences. Each sentence was evaluated by 30 participants on Amazon’s Mechanical Turk. Results Over half (52.7%) change to a more frequent frequency bin after simplification and 22.3% stayed the same. The expert writer was able to transform sentences into more frequent structures. The simplified sentences also appeared easier; perceived difficulty increased from 2.13 to 2.35. However, the sentences were not any easier to understand and the Cloze score did not change. Figure 1 shows the scores aggregated by bin. We conclude that writers need more direction to simplify text and are testing similarity functions to guide writers towards simpler structures. Figure 1. Actual difficulty scores (left, multiple choice Cloze test) and perceived difficulty (right, 5-point Likert) forthe original sentences ,“Original”, and medical writer simplified, “Simplified”. AcknowledgementsResearch reported in this publication was supported by the National Institutes of Health (NIH) #R01LM011975. Thecontent is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.References1. Gondy Leroy, James Endicott, David Kauchak, Obay Mouradi and Melisa Just. User Evaluation of the Effectsof a Text Simplification Algorithm Using Term Familiarity on Perception, Understanding, Learning, andInformation Retention. In Journal of Medical Internet Research, 2013.0.820.870.920.97 0 1 2 3 4 5 6 7 8 9 10Percent correct